Methods and Systems for Human Imperceptible Computerized Color Transfer

ABSTRACT

The present disclosure includes systems and methods for color transfer. The method includes receiving a target image, and determining dominant source colors. The method further includes transforming the target image into a color model including a target luminance component and a target color information component. Additionally, the method includes segmenting the target image into a plurality of target segments based on the target color information component or the target luminance component and extracting dominant target colors from the target image by extracting information for at least one of the dominant target colors from each target segment of the plurality of target segments. Further, the method includes generating a color mapping relationship between the dominant target colors and the dominant source colors, and creating a recolored target image using the color mapping relationship.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application62/384,419, filed Sep. 7, 2016 and entitled “Human Visual System-BasedColor Transfer System, Method and Apparatus”, the entire contents ofwhich are incorporated by reference herein.

BACKGROUND OF THE INVENTION

Various aspects of the present disclosure relate generally to methodsand systems relating to image and video analytics, object recognition,and more particularly to coloring for grayscale images, recoloring forcolor images and videos, and for generating recolored image sequences.

Many professional image/video editing software algorithms and packageshave been developed to achieve different image processing for multimediaapplications. Color transfer is one popular application to change thecolors in an image sequence. However, these professional image editingsoftware tools require the users to process most tasks manually,resulting a significant investment in labor and time. For example, tochange the color of an object in an image, a user must select the objectby drawing/selecting the boundaries of the object. Next, the user mustselect a desired color from a color palette, and “repaint” (recolor) theselected object. Accordingly, manual color transfer technology islimited, time-intensive, and user-intensive.

Therefore, what is needed is an improved color transfer system andmethod for coloring images and videos and generating a sequence of colorimages and videos.

SUMMARY OF THE INVENTION

The foregoing needs are met by the methods and systems for transferringcolor according to the present disclosure.

As stated above, various aspects of the present disclosure may generallyrelate to image and video analytics and object recognition. Inparticular, the present disclosure includes systems and methods forcoloring grayscale images, recoloring color images and videos, andgenerating new recolored image sequences. Applications of the systemsand methods may include, but are not limited to, consumer products,healthcare, the medical industry (e.g. whole sliding imaging andhistology), the fashion industry, the military (e.g. camouflage design),security (e.g. facial recognition, location recognition), databasestandardization, interior design, biometrics, medical aiding (e.g. colorblindness support and dyslexia (or other learning disorder) support),animation simulation, artwork recoloring, publishing, and othercolor-transfer technology.

Aspects of the present disclosure generally relate to image processing,and particularly to color transfer. As one non-limiting example, thepresent disclosure may be implemented in fashion design, andspecifically to explore different color effects or material effects ondesign. As another non-limiting example, the present disclosure may beused to create camouflage design (e.g. for the military). In anothernon-limiting example, systems and methods in accordance with the presentdisclosure may be used in display devices for individuals with colorvision deficiency, as well as support systems for people affected bycolor blindness.

In one aspect, the present disclosure includes a method of transferringcolor to recolor a target image. The method includes receiving a targetimage, and determining dominant source colors. The method furtherincludes transforming the target image into a color model including atarget luminance component and a target color information component.Additionally, the method includes segmenting the target image into aplurality of target segments based on the target color informationcomponent or the target luminance component and extracting dominanttarget colors from the target image by extracting information for atleast one of the dominant target colors from each target segment of theplurality of target segments. Further, the method includes generating acolor mapping relationship between the dominant target colors and thedominant source colors, and creating a recolored target image using thecolor mapping relationship.

In another aspect, the present disclosure includes a method for coloringan input grayscale image into an output color image. The method includesselecting dominant source colors from a color palette or one color froma source image. The method further includes applying a color modeltransformation to transform a target image in an original color modelinto a color model wherein a target luminance component and a targetcolor information component are independent. Additionally, the methodincludes dividing the target image into a plurality of target regionsaccording to the target luminance component. The method further includesgenerating a color mapping relationship between at least one dominanttarget color from each of the plurality of target regions and a dominantsource color, and transferring dominant source color information into atarget image. The method additionally includes applying an inverse colormodel algorithm to transfer the color model to a selected color model.

In another aspect, the present disclosure includes a method forrecoloring an input image into an output image with another colorappearance. The method includes selecting dominant source colors. Themethod further includes applying a color model transformation algorithmto transform a target image in an original color model into a colormodel wherein a target luminance component and a target colorinformation component are independent. Additionally, the method includesdividing the target image into a plurality of target segments accordingto the target color information component, and extracting dominanttarget colors from the target image by extracting information for atleast one of the dominant target colors from each target segment of theplurality of target segments. The method further includes generating acolor mapping relationship between the dominant target colors and thedominant source colors, and transferring source color information into atarget image based on information generated from a source color inputalgorithm. The method additionally includes applying an inverse colormodel algorithm to transfer the color model to a selected color model.

In another aspect, the present disclosure includes a method for imagesegmentation by data grouping. The method further includes receiving anoriginal image and setting a number of segment groups manually orautomatically via a computer algorithm. Additionally, the methodincludes applying a color model transformation algorithm to transformthe original image in an original color model into a color model whereina target luminance component and a target color information componentare independent. The method additionally describes including the targetcolor information component as a feature for each pixel in the originalimage. Further, the method includes grouping the pixels via aLogarithmic GMM method, using each target color information component.

In another aspect, the present disclosure includes a method for imagesegmentation by data grouping. The method includes receiving an originalimage, and setting a number of segment groups manually or automaticallyvia a computer algorithm. The method further includes applying a colormodel transformation algorithm to transform the original image in anoriginal color model into a color model wherein a target luminancecomponent and a target color information component are independent.Additionally, the method describes including the target colorinformation component as a feature for each pixel in the original image.Further, the method includes grouping the pixels via a LogarithmicK-means method, using each target color information component.

In another aspect, the present disclosure includes a method forgenerating an image sequence showing a gradual changing from a firstcolor appearance to a second color appearance. The method includesdetermining at least two sets of dominant source colors. The methodfurther includes applying a color model transformation algorithm totransform a target image in a first color model into a color modelwherein a target luminance component and a target color informationcomponent are independent. Additionally, the method includes segmentingthe target image into a plurality of target segments according to thetarget color information component or the target luminance component.The method further includes extracting dominant target colors from thetarget image by extracting information for at least one of the dominanttarget colors from each target segment of the plurality of targetsegments. Additionally, the method includes generating a color mappingrelationship between the dominant target colors and the at least twosets of dominant source colors. The method includes calculating colorinformation for probabilistic color transfer via a color variationcurve, and transferring the color information into a target image byusing the color information generated from the color variation curve.The method further includes applying an inverse color model algorithm totransfer the first color model to a selected second color model.

In another aspect, the present disclosure includes a support system forcolor-impaired users. The system includes a pair of glasses configuredto be worn by a color-impaired user. The system further includes atleast one camera affixed to the pair of glasses, and a processor incommunication with the at least one camera. The processor is configuredto capture at least one image via the at least one camera, determinedominant source colors, and segment a target image into a plurality oftarget segments based on a target color information component. Theprocessor is further configured to extract dominant target colors fromthe target image by extracting information for at least one of thedominant target colors from each target segment of the plurality oftarget segments. Additionally, the processor is configured to generate acolor mapping relationship between the dominant target colors and thedominant source colors, and transfer color information into the targetimage. The processor is additionally configured to generate images forthe color-impaired user, and display the generated images on at leastone lens of the pair of glasses.

In another aspect, the present disclosure includes a method for testingthe performance of biometrics recognition technology. The methodincludes receiving a biometrics image. The method further includesdetermining dominant source colors, and segmenting the biometrics imageinto a plurality of biometric segments based on a biometrics colorinformation component. Additionally, the method includes extractingdominant target colors from the biometric image by extractinginformation for at least one of the dominant target colors from eachbiometric segment of the plurality of biometric segments. Further, themethod includes generating a color mapping relationship between thedominant target colors and the dominant source colors, and transferringcolor information into the biometrics image. Additionally, the methodincludes extracting at least one biometrics feature from the biometricsimage. The method includes comparing the at least one biometrics featurewith a reference data set, and generating a test result.

In another aspect, the present disclosure includes a method for coloringan input grayscale image into an output colorful image. The methodincludes applying a color model transformation algorithm to transform atarget image in an original color model into a color model wherein atarget luminance component and a target color information component areindependent. Additionally, the method includes segmenting a target imageinto a plurality of target segments based on the target luminancecomponent. Further, the method includes extracting structure featuresfrom the target image by extracting information for at least one of thestructure features from each target segment of the plurality of targetsegments. The method includes generating a source color for each targetsegment based on each structure feature, via a machine learning model,and transferring the dominant source colors into the target image via acopy process. The method further includes applying an inverse colormodel algorithm to transfer the original color model to a selectedsecond color model.

In another aspect, the present disclosure includes a method for partialcolor transfer. The method includes selecting an object to be colortransferred, the object included in a target image. Additionally, themethod includes determining dominant source colors, and transforming thetarget image from an original color model into a color model including atarget luminance component and a target color information component. Themethod further includes segmenting the object into a plurality of objectsegments based on the target color information component or the targetluminance component. Additionally, the method includes extractingdominant target colors from the object by extracting information for atleast one of the dominant target colors from each object segment of theplurality of object segments. The method further includes generating acolor mapping relationship between the dominant target colors and thedominant source colors. Additionally, the method includes transferringcolor information into the object, and applying an inverse color modelalgorithm to transfer the original color model to a selected colormodel.

In another aspect, the present disclosure includes a method for partialcolor transfer in a video. The method includes inputting at least oneobject to be color transferred, the at least one object included in avideo. The method further includes detecting the at least one object ineach frame image of the video, and determining dominant source colors.Additionally, the method includes transforming each frame image from anoriginal color model into a color model including a frame luminancecomponent and a frame color information component. Further, the methodincludes segmenting the at least one object into a plurality of objectsegments based on the frame color information component or the frameluminance component. Additionally, the method includes extractingdominant target colors from the at least one object by extractinginformation for at least one of the dominant target colors from eachobject segment of the plurality of object segments. The method includesgenerating a color mapping relationship between the dominant targetcolors and the dominant source colors, and transferring colorinformation into the at least one object in each frame image. The methodadditionally includes applying an inverse color model algorithm totransfer the original color model to a selected color model.

In another aspect, the present disclosure includes a method ofrecoloring text for people with a learning disability. The methodincludes detecting and extracting text from a target image. The methodfurther includes transforming the target image from an original colormodel into a color model including a target luminance component and atarget color information component. Additionally, the method includesdetermining dominant source colors. Further, the method includestransferring color information into the text via probabilistic colortransfer. The method additionally includes applying an inverse colormodel algorithm to transfer the original color model to a selected colormodel.

The foregoing and other aspects of the invention will appear from thefollowing description. In the description, reference is made to theaccompanying drawings which form a part hereof, and in which there isshown by way of illustration a preferred aspect of the invention. Suchaspect does not necessarily represent the full scope of the invention,however, and reference is made therefore to the claims and herein forinterpreting the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will hereafter be described with reference to theaccompanying drawings, wherein like reference numerals denote likeelements.

FIG. 1 is a flow diagram illustrating a non-limiting HIC-based colortransfer method according to the present disclosure.

FIG. 2A is a flow diagram illustrating a sub-method of FIG. 1corresponding to inputting a source color.

FIG. 2B is a non-limiting example system implementing the sub-method ofFIG. 2A.

FIG. 3 is a flow diagram illustrating a sub-method of FIG. 1corresponding to transforming via a color model.

FIG. 4 is a flow diagram illustrating a sub-method of FIG. 1corresponding to dividing an image.

FIG. 5 is a flow diagram illustrating a sub-method of FIG. 1corresponding to building a color mapping.

FIG. 6 is a non-limiting example of the color mapping sub-method of FIG.5.

FIG. 7 is a flow diagram illustrating a sub-method of FIG. 1corresponding to performing a color transfer.

FIG. 8 is a flow diagram illustrating a sub-method of FIG. 1corresponding to transforming via an inverse color model.

FIG. 9 is a flow diagram of a non-limiting example HIC-based colortransfer system and method according to the present disclosure.

FIG. 10 is a flow diagram of another non-limiting example HIC-basedcolor transfer system and method, using a source image, according to thepresent disclosure.

FIG. 11 is a flow diagram of a non-limiting example HIC-based colortransfer sequence system and method according to the present disclosure.

FIG. 12 is a flow diagram of another non-limiting example HIC-basedcolor transfer sequence system and method according to the presentdisclosure.

FIG. 13 is a flow diagram of a non-limiting example color-blindHIC-based color transfer system and method according to the presentdisclosure.

FIG. 14 is a flow diagram of another non-limiting example color-blindHIC-based color transfer system and method according to the presentdisclosure.

FIG. 15 is a flow diagram of another non-limiting example HIC-basedcolor transfer system and method according to the present disclosure.

FIG. 16 is a flow diagram of another non-limiting example HIC-basedcolor transfer system and method according to the present disclosure.

FIG. 17 is a flow diagram illustrating a non-limiting video HIC-basedcolor transfer method according to the present disclosure.

FIG. 18 is a flow diagram of a non-limiting example video HIC-basedcolor transfer system and method according to the present disclosure.

FIG. 19 is a flow diagram of a non-limiting example textile HIC-basedcolor transfer system and method according to the present disclosure.

FIG. 20 is a flow diagram of a non-limiting example texture HIC-basedcolor transfer system and method according to the present disclosure.

DETAILED DESCRIPTION

Before the present invention is described in further detail, it is to beunderstood that the invention is not limited to the particular aspectsdescribed. It is also to be understood that the terminology used hereinis for the purpose of describing particular aspects only, and is notintended to be limiting. The scope of the present invention will belimited only by the claims. As used herein, the singular forms “a”,“an”, and “the” include plural aspects unless the context clearlydictates otherwise.

It should be apparent to those skilled in the art that many additionalmodifications beside those already described are possible withoutdeparting from the inventive concepts. In interpreting this disclosure,all terms should be interpreted in the broadest possible mannerconsistent with the context. Variations of the term “comprising”,“including”, or “having” should be interpreted as referring to elements,components, or steps in a non-exclusive manner, so the referencedelements, components, or steps may be combined with other elements,components, or steps that are not expressly referenced. Aspectsreferenced as “comprising”, “including”, or “having” certain elementsare also contemplated as “consisting essentially of” and “consisting of”those elements, unless the context clearly dictates otherwise. It shouldbe appreciated that aspects of the disclosure that are described withrespect to a system are applicable to the methods, and vice versa,unless the context explicitly dictates otherwise.

As used herein, the term “target image” generally refers to an image tobe color transformed. Also used herein, the term “source image”generally refers to an image containing color information to betransferred. In a general sense, color elements of the source image maybe used to recolor a target image (e.g. an input image), resulting in arecolored output image. As used herein, the term “artifacts” generallyrefers to unnatural image contents which may appear fake. As usedherein, the term “Human Imperceptible Computerized System” (“HIC-basedSystem”) generally refers to a model of how humans perceive images in anatural manner.

Aspects of the present disclosure are explained in greater detail in thedescription that follows. Aspects of the disclosure that are describedwith respect to a method are applicable to aspects related to systemsand other methods of the disclosure, unless the context clearly dictatesotherwise. Similarly, aspects of the disclosure that are described withrespect to a system are applicable to aspects related to methods andother systems of the disclosure, unless the context clearly dictatesotherwise.

As previously described, traditional color transfer technology can belimited, time-intensive, and user-intensive. Existing color transfermethods can generally be classified as: (1) global color (GC) transfermethods, and (2) local color (LC) transfer methods.

A global color (GC) transfer is a transformation that does not considerthe color correspondences and spatial relations of different coloredregions. The idea of global color transfer was derived from E. Reinhard,M. Adhikhmin, B. Gooch, and P. Shirley, “Color transfer between images,”IEEE Computer Graphics and Applications, vol. 21, pp. 34-41, 2001. Astatistical color transfer algorithm was developed based on standarddeviation and mean value to make the entire target image transform tothe same color contained within the source image.

The GC transfer method suffers from the problems of artifacts when thesource and target images have substantially different colordistributions. Output images from the GC transfer method generally donot have acceptable spatial relationships when the target image containsdifferent color regions. When the target image contains two dominantcolors (such as “blue sky” and “yellow earth”), the GC algorithm willtransfer both of dominant colors to one color (such as red).Accordingly, this makes the output recolored image appear artificial,especially to the human eye. Further, the GC transfer cannot distinguishdifferent image statistics and will frequently mix up image regions.

Different from GC transfer, a local color (LC) transfer takes colorcorrespondences and spatial relationships into consideration. Theresearch about local color transfer can be grouped into threecategories: (1) methods with human, (2) methods with segmentation, and(3) example-based methods.

The first category of LC transfer requires human interaction. As oneexample, Liu et al. introduced a local color method that requires usersto set a window to select the color that the algorithm should change.This is disclosed in S. Liu, H. Sun, and X. Zhang, “Selective colortransferring via ellipsoid color mixture map,” Journal of VisualCommunication and Image Representation, vol. 23, pp. 173-181, 1 2012.Then, using the statistical information of the selected color of thetarget image, the Reinhard's color transfer algorithm changes the color.

Using an ellipsoid color mixture map, the content with a similar colorto the one in the selection window will be changed, while the othercolor contents are kept the same. In another LC method, strokes areutilized as a method of interface to indicate what color each color inthe target image will be recolored to. This method is discussed in C.-L.Wen, C.-H. Hsieh, B.-Y. Chen, and M. Ouhyoung, “Example-based MultipleLocal Color Transfer by Strokes,” Computer Graphics Forum, vol. 27, pp.1765-1772, 2008. However, these LC methods are still used to do onesingle color transfer at one time.

To solve the “one color” problem discussed above, a second category oflocal color transfer methods use a region segmentation procedure toretain spatial coherence and extract the dominant color information.This method is discussed in T. Yu-Wing, J. Jiaya, and T. Chi-Keung,“Local color transfer via probabilistic segmentation byexpectation-maximization,” in Computer Vision and Pattern Recognition,2005. CVPR 2005. IEEE Computer Society Conference on, 2005, pp. 747-754vol. 1.

Tai et al. previously utilized an expectation-maximization scheme to doprobabilistic segmentation for local color transfer. Using the colorinformation of each segmentation from the target image and source image,the color transfer is processed individually to recolor eachsegmentation.

Another LC transfer method using dominant colors is discussed in J.-D.Yoo, M.-K. Park, J.-H. Cho, and K. H. Lee, “Local color transfer betweenimages using dominant colors,” Journal of Electronic Imaging, vol. 22,pp. 033003-033003, 2013. Here, the dominant color information is firstextracted from both the target image and the source image. Next, amapping function is constructed for each dominant color in the targetimage, where each target color is mapped to one of the dominant colorsin the source image. In this procedure, the method requires anadditional adjustment process for the color transfer, since doingstatistical multiple color transfer will generate artificial edges.Summarily, the core statistical color transfer is straight-forward, buthas difficulties with edges and tends to change the luminancedistribution (e.g. a dark color is undesirably transferred into alighter color).

Accordingly, there are two main drawbacks for the first two kinds ofcolor transfer algorithms: (1) artifacts (fake edges), and (2) undesiredluminance changes.

Additionally, some existing algorithms will generate fake informationafter the color transfer. This results from the similar color pixels inthe target image that are then mapped to highly different colors. In thetarget image, the color changes gradually and thus there is no evidentcolor edge. However, after recoloring, noticeable fake edge informationis generated.

The luminance change problem results from some existing color transferalgorithms doing the color transfer in the luminance plane. This makesthe target image recolored to some output images with poor luminance andloss of content information. In many cases, the loss of content isapparent.

The third category of LC transfer, example-based methods, is discussedin A. Bugeau, T. Vinh-Thong, and N. Papadakis, “VariationalExemplar-Based Image Colorization,” Image Processing, IEEE Transactionson, vol. 23, pp. 298-307, 2014 as well as X. Yi, W. Liang, L. Chi-Sing,L. Yu-Kun, and W. Tien-Tsin, “Example-Based Color Transfer for GradientMeshes,” IEEE Transactions on Multimedia, vol. 15, pp. 549-560, 2013.The idea behind these example-based algorithms is that image sectionswith similar structure information should share similar colorinformation. Hence, for each section in the target image, the algorithmssearch all the sections in the source image to find the sections with asimilar structure, and accordingly utilizes the color information ofthese similar sections to execute the coloring/recoloring. However,these methods have three main drawbacks. First, the source image mustshare similar content with the target image (e.g. both images arelandscape images). Second, the sections in the source image havingsimilar structure information must have a similar color. Third, thesemethods typically require high computational cost.

The present disclosure overcomes the above described problems associatedwith traditional color transfer systems and methods. The presentdisclosure includes Human Imperceptible Computerized (HIC-based) colortransfer systems and methods for, or for use in, coloring grayscaleimages and videos, recoloring color images and videos, and generatingsequences of color images and videos.

In some aspects, the present disclosure provides improved techniques andcomputer algorithms for coloring a grayscale image with source colors.The present disclosure may turn one grayscale image into colorfulimages. The present disclosure may recolor a colorful target image tohave another color distribution. As a non-limiting example, a picture ofa tree in summer may be recolored to appear as an autumn tree.

In another aspect, the present disclosure provides improved techniques,and computer algorithms for generating an image sequence with an inputimage and more than one source image. The image sequences generated bythe systems/methods depict the change of source color. As onenon-limiting example, the image sequences can present a tree picturetaken in summer to be one colored as in spring, then change to onecolored as in summer, followed by autumn and by winter. This exampleimage sequence shows gradual color changes in the tree picture. Asanother non-limiting example, a clothing designer can explore thedifferent color effects in a particular design by generating a designimage sequence—as opposed to manufacturing the design in each desiredcolor.

In one aspect of the present disclosure, the systems and methodsconsider the properties of human perception, including non-linearsensitivity to different levels of luminance. Accordingly, the disclosedarithmetic operations are processed and replaced by the parametricallogarithmic image processing (PLIP) operations. It has been shown thatthe key mathematical properties of the logarithmic image processingcalculus have similarities with Human Imperceptible Computerized (KPanetta, S Agaian, Y Zhou, E J Wharton, Parameterized logarithmicframework for image enhancement, IEEE Transactions on Systems, Man, andCybernetics, Part B (Cybernetics) 41 . . . pages 460-473, 2011; EWharton, S Agaian, K Panetta Comparative study of logarithmicenhancement algorithms with performance measure, Electronic Imaging2006, 606412-606412-12).

The present disclosure may utilize the following five aspects: (1) aprobabilistic color transfer model, (2) an automatic multi-color mappingalgorithm, (3) a generalized HIC-based color transfer system, (4) aparametrical logarithmic image processing operation, and (5) aLogarithmic Gaussian Mixture Model. Unlike existing methods whichtransfer the color pixel-by-pixel using single pieces of colorinformation from the source image, the presently disclosed HIC-basedcolor transfer systems and methods aim to transfer color in a targetimage using a combination of different pieces of color informationobtained from a source image. The weight of each piece of colorinformation within the combination may be calculated by a probabilitymatrix of the processed color pixel in the target image belonging toeach corresponding section. The presently disclosed model addresses thechallenging issue of target image areas that exhibit gradual colorchanges.

All the traditionally used arithmetic operations (including addition,subtraction, and multiplication) may be replaced by the parametricallogarithmic image processing (PLIP) operations. This replacement mayenable the human imperceptible computerized system's abilities todistinguish between useful and extraneous data. Weber-Fechner'sLogarithmic Law asserts that the human eye is sensitive to the intensityvariation between an object and the background, divided by thebackground intensity. This Law quantifies the minimum change that isregarded as contrast in human imperceptible computerized systems.

Different background luminance has the minimum change needed to bedetected as contrast in human imperceptible computerized systems.Background luminance ranges may be divided into three regions toapproximate closely. To make the image processing more accurate withrespect to human imperceptible computerized systems, the PLIP model wasintroduced by K. Panetta, S. Agaian, Y. Zhou and E. J. Wharton,“Parameterized Logarithmic Framework for Image Enhancement,” in IEEETransactions on Systems, Man, and Cybernetics, Part B (Cybernetics),vol. 41, no. 2, pp. 460-473, April 2011. The PLIP arithmetic operationsare shown below in Table I. There are four parameters that can be setand trained for a PLIP system. When ∈(E), ψ(E), and γ(E) all approachinfinity, the PLIP operators approach the traditional arithmeticoperations.

TABLE I PLIP Arithmetic Operation Traditional Arithmetic Operations PLIPOperations g(i, j) = ϵ(E) − f (i, j) Addition${g_{1}\overset{\sim}{\oplus}g_{2}} = {g_{1} + g_{2} - \frac{g_{1}g_{2}}{\gamma (E)}}$Substraction${g_{1}\overset{\sim}{\ominus}\; g_{2}} = {{\psi (E)}\frac{g_{1} - g_{2}}{{\psi (E)}g_{2}}}$Multiplication with constant (c)${c\overset{\sim}{\otimes}\; g_{1}} = {{\gamma (E)} - {{\gamma (E)}\left( {1 - \frac{g_{1}}{\gamma (E)}} \right)^{C}}}$Summation${\sum\limits_{1 \leq j \leq K}^{\sim}\; A_{j}} = \left( {{\ldots \mspace{11mu} \left( {\left( {A_{1}\overset{\sim}{\oplus}A_{2}} \right)\overset{\sim}{\oplus}A_{3}} \right)\mspace{14mu} \ldots}\; \overset{\sim}{\oplus}A_{K}} \right)$Mean value${\overset{\sim}{\mu}(A)} = {\frac{1}{H}{\sum\limits_{1 \leq j \leq H}^{\sim}A_{i}}}$Standard deviation${\overset{\sim}{\sigma}(A)} = \sqrt{\frac{1}{H}{\sum\limits_{1 \leq j \leq H}^{\sim}\left( {A_{i}\overset{\sim}{\ominus}{\overset{\sim}{\mu}(A)}} \right)^{2}}}$(f(i,j) is the original image intensity, g(i,j), g₁, g₂ are the graytone functions, parameters ∈(E), ψ(E), and γ(E) are functions of E thatcan be trained for the system)

FIG. 1 shows a non-limiting HIC-based color transfer method 1000according to the present disclosure. At process block 1050, a targetimage may function as an input into the color-transfer system. Next, atprocess block 1100, dominant source colors may be input into thecolor-transfer system. The dominant source colors may be defined by arespective color number at process block 1150. The target image may betransformed, as necessary, via a color model at process block 1200.Next, the target image may be divided into sections at process block1250. At process block 1300, a color map may be constructed. Once thecolor map is constructed, a color transfer may occur to the target imageat process block 1350. Once the color transfer is complete, the targetimage may be transformed via an inverse color model at process block1400. The process blocks shown in FIG. 1 will be described in greaterdetail below, with reference to subsequent figures.

The disclosed process for inputting dominant source colors can be brokeninto two main areas: (1) color source option, and (2) source colorextracting.

In some aspects, the color source may come from manually extracteddominant source colors. In other aspects, the color source may come fromautomatically extracted dominant source colors. More specifically, thepresent disclosure may have two options to extract dominant sourcecolors. One option may be to select the dominant source colors manually(e.g. from a color palette). Another option may be to use an automaticcomputer algorithm to extract the dominant source colors from a sourcecolor image. Users may use either option to select the source ofdominant colors.

In some aspects, when users select the option of manual extraction, thecomputer system may show a color palette, and users may choose thedominant source colors by clicking the corresponding color block in thecolor palette display (for example, a window may pop up that containsthe color palette). Then, the information of the selected dominantsource colors may be used in the probabilistic color transfer block.

In some aspects, when users select the option of automatic extraction,the computer system may need the user to upload a source color imagefile (accordingly, the system may prompt the user to upload an imagefile). For the uploaded color image file, a color-based imagesegmentation method may be applied to divide the image into differentsegments. The image segmentation may be any existing or newly designedimage segmentation method. The segmentation method may be color-basedimage segmentation. Here, the segments may be processed using segmentintegration to make the number of segments equal to a default number, ora number set by a user. In some situations, it may be beneficial todivide the target image into two or three regions. After the imagesegmentation, the dominant source color information may be calculated.This calculation may be achieved by a variety of methods, including butnot limited to, the mean value of color information in each segmentand/or the median value of color information in each segment.

FIGS. 2A and 2B show a sub-method and system corresponding to selectinga source color option and dominant source colors. Process block 1100 maybe further described by FIG. 2A. At process block 1105, a user maydecide if the selection of source colors will be automatic or manual. Ifthe user decides to do manual selection, a color palette may be shown atprocess block 1110. The user may then select a “k” number of dominantsource colors at process block 1115. After the dominant source colorsare selected, the dominant source colors may be extracted from the colorpalette at process block 1130. Next, process block 1150 may define thedominant source color numbers.

Alternatively, the user may decide to do automatic selection, and asource image may be used as an input at process block 1120. Next, thesource image may be segmented/divided at process block 1125. Oncesegmented, the dominant source colors may be extracted at process block1130. As with above, the dominant source color numbers may then bedefined at process block 1150.

FIG. 2B shows a non-limiting example demonstrating the selection of asource color option and the associated dominant source colors. Here, acolor palette may be presented to a user for color selection. Next, atable may display the color numbers a, 13 associated with the dominantsource colors. Alternatively, a selected source image may be segmentedand the table may display the color numbers associated with thosedominant source colors.

In some aspects, the present disclosure may extract a color informationcomponent and a luminance component individually. By color modeltransformation, the further probabilistic color transfer equation may beapplied to the color information component of a target image and retainthe luminance component. Dividing color information and luminance mayaddress luminance change between the target image and the output image,which is one of the drawbacks of existing color transfer algorithms.

Since the probabilistic color transfer equations may only be applied tothe color information component, the images may be processed using acolor model transformation to ensure that the color informationcomponent and the luminance component remain independent.

The color model applied after the color model transformation may be anyexisting color model, wherein color component and luminance componentare independent. These color model examples include CIELαβ, YCrCb, andany other color model where color components and luminance componentsare independent.

After being processed according to the present disclosure, acorresponding inverse color model transformation may be applied toreturn the color model of the original input target image. This inversecolor model transformation may ensure that the input and output imagesshare the same color model.

FIG. 3 shows the sub-method of FIG. 1 corresponding to transforming thetarget image via a color model. In some aspects, process block 1200 maybe further described. At process block 1205, the computer system maydetermine if the target image is a grayscale image. If the target imageis grayscale, the target image may be transferred to a color model atprocess block 1210 (for example, an RGB model). If the target image isnot grayscale, the target image may not be transferred to another colormodel. At process block 1215, the computer system may determine if thetarget image has independent luminance and color information. If thetarget image does not have independent luminance and color information,then the target image may be transformed to a component model at processblock 1220. The component model may have independent luminance and colorinformation components. If the target image already has independentluminance and color information, a transformation as described may notoccur.

In some aspects, the process for image segmentation may be broken intofour categories: (1) logarithmic K-means method, (2) logarithmic GMMmethod, (3) Logarithmic K-means method with missing Data, and (4) Othermethods to split image into similar segments.

For a grayscale input target image, the image segmentation may be anyexisting or new method. For a grayscale input target image, the imagesegmentation may only utilize the luminance information of the targetimage.

For a color input target image, the image segmentation may be anyexisting or new method. For a color input target image, the imagesegmentation may only utilize the color information of the target image.

These image segmentations may be implemented by data classification andother image segmentation methods, such as graph-oriented methods,mean-shift algorithms, Watershed segmentation, and partial differentialequation-based methods. Here, two examples are provided. However, theimage segmentation is not limited to these two examples. In thefollowing non-limiting example, the number of segments after imagesegmentation is equal to the number of dominant colors, which may bemanually chosen or automatically decided via algorithms.

In some aspects, a logarithmic K-means method may be used to segment theimage. One way to do image segmentation may be to utilize existingclustering techniques in machine learning, such as K-means. Theclassical K-means may be utilized to set large sample data intodifferent K clusters. This classical K-means may be improved to be alogarithmic K-means method.

In one aspect of the present disclosure, logarithmic K-means may be usedto extract the domain colors of a source image. The method input mayinclude the sample data matrix and K setting. The number of similarsegments may be selected manually or automatically. The output of thelogarithmic K-means method is an index matrix with the same size ofsample data and in the range of [1, K]. The algorithm of logarithmicK-means may be shown in an iteration process of three steps.

In the first step, for a given cluster assignment C, find theobservation in the cluster minimizing the total distance to other pointsin that cluster:

$\begin{matrix}{i_{k}^{*} = {\underset{\{{{i\text{:}{C{(i)}}} = k}\}}{\arg \mspace{11mu} \min}\mspace{14mu} {\sum\limits_{{C{(j)}} = k}^{\sim}{d\left( {x_{i},x_{j}} \right)}}}} & (1)\end{matrix}$

Where, C is the two-dimensional data with size of 2*(w_(S)l_(S)) byindividually scanning the pixel values in α and β planes of colorfulsource image with size of w_(S)*l_(S) from left to right and up to down.d(x_(i),x_(j)) is any distance measure. As an example here, it may be aEuclidean distance.

d(x _(i) ,x _(j))=∥x _(i) {tilde over (⊖)}x _(j)∥²  (2)

Where, m_(k)=x_(i*) _(k) , k=1, 2, . . . , K

In step 3, for a set of cluster centers {m₁, . . . , m_(k)}, the totalerror can be minimized by assigning each observation to the closest(current) cluster center:

$\begin{matrix}{{C(i)} = {\underset{1 \leq k \leq K}{\arg \mspace{11mu} \min}\mspace{14mu} {d\left( {x_{i},m_{k}} \right)}}} & (3)\end{matrix}$

After repeating steps 1, 2, and 3, the Logarithmic K-means method is tofind:

$\begin{matrix}{\underset{S}{\arg \mspace{11mu} \min}{\sum\limits_{1 \leq i \leq K}^{\sim}{\sum\limits_{ \in S_{i}}^{\sim}{{\overset{\sim}{\ominus}{\overset{\sim}{\mu}}_{i}}}^{2}}}} & (4)\end{matrix}$

Where,

=[

₁,

₂, . . . ,

_(n)] is the large sample date, which will be grouped into K sets S=[S₁,S₂, . . . , S_(k)].

In one aspect of the present disclosure, a logarithmic Gaussian MixtureModel (GMM) method may be utilized for data clustering to achieve imagesegmentation.

To consider the non-linearity of human imperceptible computerizedsystems, the Logarithmic Gaussian Mixture Model is proposed as avariation of Gaussian Mixture Model with Parameterized LogarithmicFramework. By replacing the linear arithmetic operations (addition,subtraction, and multiplication) with nonlinear parameterizedlogarithmic image processing (PLIP) operations, the disclosedLogarithmic GMM method accurately characterizes the nonlinearity ofcomputer image. The Logarithmic Gaussian Mixture Model is defined as aweighted PLIP sum of the individual Logarithmic Gaussian probabilitydensity function:

$\begin{matrix}{{p\left( C_{i} \right)} = {\sum\limits_{1 \leq j \leq K}^{\sim}{\alpha_{j}\overset{\sim}{\otimes}{g\left( {C_{i},{\overset{\sim}{\mu}}_{j},{\overset{\sim}{\sigma}}_{j}} \right)}}}} & (5)\end{matrix}$

Where, α₁ are the mixture weight trained by the system.

$\begin{matrix}{{g\left( {T_{i}^{1},{\overset{\sim}{\mu}}_{j},{\overset{\sim}{\sigma}}_{j}} \right)} = {\frac{1}{\sqrt{2\pi {\overset{\sim}{\sigma}}_{j}^{2}}}\exp \mspace{11mu} \left( \frac{- \left( {T_{i}^{1}\overset{\sim}{\ominus}{\overset{\sim}{\mu}}_{j}} \right)^{2}}{2{\overset{\sim}{\sigma}}_{j}^{2}} \right)}} & (6)\end{matrix}$

Where, {tilde over (μ)}_(j) is the PLIP mean value of each region and{tilde over (σ)}_(j) is the PLIP standard deviation of each region.

The algorithm of Logarithmic GMM may be described by five steps. In thefirst step, initialize the PLIP means {tilde over (μ)}_(j), standarddeviation {tilde over (σ)}_(j), and mixture weight α₁. In the secondstep (called E step), the responsibilities are evaluated by using thecurrent parameter values and Logarithmic Gaussian mixture model in Eqn.(5). In the third step (called M step), the parameters will bere-estimated using the current responsibilities.

$\begin{matrix}{{\overset{\sim}{\mu}}_{j}^{new} = {\frac{1}{N_{j}}{\sum\limits_{1 \leq i \leq {w_{S}l_{S}}}^{\sim}{{p\left( C_{i} \right)}\overset{\sim}{\otimes}C_{i}}}}} & (7) \\{{\overset{\sim}{\sigma}}_{j}^{new} = \sqrt{\frac{1}{N_{j}}{\sum\limits_{1 \leq i \leq {w_{S}l_{S}}}^{\sim}{{p\left( C_{i} \right)}\overset{\sim}{\otimes}{{C_{i}\overset{\sim}{\ominus}{\overset{\sim}{\mu}}_{j}^{new}}}^{2}}}}} & (8) \\{\alpha_{j}^{new} = \frac{N_{j}}{w_{S}l_{S}}} & (9) \\{{Where},} & \; \\{N_{j} = {\sum\limits_{1 \leq i \leq {w_{S}l_{S}}}^{\sim}{p\left( C_{i} \right)}}} & (10)\end{matrix}$

In the fourth step, the log likelihood will be evaluated.

$\begin{matrix}{{\ln \mspace{11mu} {p\left( {{C\alpha},\overset{\sim}{\mu},\overset{\sim}{\sigma}} \right)}} = {\sum\limits_{1 \leq i \leq {w_{S}l_{S}}}^{\sim}\mspace{11mu} {\ln \mspace{11mu} \left\{ \; {\sum\limits_{1 \leq j \leq K}^{\sim}{\alpha_{j}\overset{\sim}{\otimes}{g\left( {C_{i},{\overset{\sim}{\mu}}_{j},{\overset{\sim}{\sigma}}_{j}} \right)}}} \right\}}}} & (11)\end{matrix}$

In the fifth step, if the standard variation of either the parameters orthe log likelihood is satisfied by the pre-setting threshold, thisalgorithm will end. Otherwise, it will go back to step 2 (E step).

In many real applications, missing data is common. To do the imagesegmentation with some missing data, a new K-means method with partiallyobserved data (K-POD) was proposed in Jocelyn T. Chi, Eric C. Chi, andRichard G. Baraniuk, “k-POD: A Method for k-Means Clustering of MissingData,” The American Statistician, Vol. 70, Iss. 1, 2016. In the same wayas improving K-means and GMM via the present disclosure, K-POD may beimproved to Logarithmic K-POD by replacing some arithmetic operationswith PLIP operation.

In another aspect of the present disclosure, other methods to achieveimage segmentation include grouping by matching to find similar segmentsto the reference segment. The users or computer may select the targetsegment to process manually or automatically in the target image. Then,by searching through the whole image, other similar segments will befound. The similar segments may share similar structural or statisticalinformation. After searching all the segments in the image, all thesimilar segments to the target segment may be extracted and processedfurther.

For data classification methods to achieve image segmentation, thenumber of segments in an input target image may be preset by users. Forother image segmentation methods, the number of segments aftersegmentation may vary and, in some situations, be larger than a user'ssetting. Accordingly, a segment integration process may be introduced tointegrate the segments and thus reduce the number of segments andcorresponding dominant target colors. The algorithm of segmentintegration may be processed to combine close dominant target colors asone to reduce the number of segments until it reaches the set number ofsegments.

For each region in the target image, some dominant color information maybe collected for subsequent steps. In some aspects, the dominant colorinformation may be the mean value and standard deviation of eachsegment. However, dominant color information may not be limited to theseprevious two types. As an example, the present disclosure may also usemedian value or tri-median value.

FIG. 4 shows a sub-method of FIG. 1 corresponding to segmenting/dividingan image. In some aspects, process block 1250 may be further described.At process block 1255, the computer system determines if the targetimage is grayscale. If the target image is grayscale, the target imagemay be segmented based on luminance information at process block 1260.If the target image is not grayscale, the target image may be segmentedbased on color information at process block 1265.

Still referring to FIG. 4, regions of the target image may be integrated(according to the above description) at process block 1270. Onceintegrated, the dominant color information may be extracted from thetarget image at process block 1275. Then, a possibility matrix may becalculated at process block 1280.

To achieve the probabilistic color transfer, possibility matrixgeneration may provide the possibility information of each pixel in atarget image. Here, various probabilistic models may be utilized tocalculate the possibility of each pixel belonging to each segment afterimage segmentation. These various probabilistic models may be theGaussian distribution model, the Beta distribution model, the Log-normaldistribution model, the Dirac delta function, the truncated normaldistribution model, and other possibility distribution models. When thepossible distribution model is the Bernoulli distribution model, thiscolor transfer algorithm will be the traditional color transferalgorithm in E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, “Colortransfer between images,” IEEE Computer Graphics and Applications, vol.21, pp. 34-41, 2001.

As a non-limiting example, the Gaussian distribution model may beutilized to calculate the percentage of each pixel belonging to eachsegment. This percentage vector matrix has the size of k*(w_(T)l_(T)) asshown in Eqn. (5).

P(j,i)=g(T _(i) ¹,{tilde over (μ)}_(j),{tilde over (σ)}_(j))  (12)

The present disclosure's process for color mapping is described below.Users may manually build the color mapping relationship between dominantsource colors and dominant target colors. Alternatively, the system mayautomatically create a color mapping relationship based on theinformation of dominant source colors and dominant target colors.

For the manual method, in one aspect, users may manually associatedominant source colors and dominant target colors in a color mappingwindow to indicate the color mapping relationship. In some situations,it may be beneficial to have the user draw lines (e.g. using thecomputer system) between desired dominant source colors and dominanttarget colors.

The present disclosure includes an automatic method. Traditional mappingfunctions generally have the problem of limited usability of colors fromthe source images. This often means that the colors in a target imageare all mapped to one of the colors in the source image. Accordingly,this results in some colors in the source image not appearing in thetarget image, thus making the color transfer unsuccessful.

To solve the above problem, the automatic color mapping of the presentdisclosure may be a bidirectional mapping (

) function, which may find the extracted dominant colors from thesource/target image for each dominant color of the target/source image.The optimal

can be found by Eqn. (13).

arg   min  ∑ 1 ≤ i ≤ K ∼   F  ( C S  ( i ) )  ⊖ ~  F  ( C T i)  2 ( 13 )

Where, F(C_(T) ^(i)) is a vector for each dominant color in the targetimage, including two normalized components: (1) ratio of each dominantcolor, and (2) order of mean distance from each dominant color to therest of the dominant colors. These orders may be divided by K to be inthe same range with ratios of each domain color.

FIG. 5 shows a sub-method of FIG. 1 corresponding to building a colormapping. Process block 1300 may be further described. At process block1305, the computer system may determine if the source color has beenautomatically input or manually (e.g. by the user). If the source colorswere input manually, a color mapping matrix may be input at processblock 1310. If the source colors were input automatically, percentageand distance information (as described above), may be calculated atprocess block 1315. Once calculated, a color mapping matrix may be builtat process block 1320.

FIG. 6 shows a non-limiting example of a color mapping matrix asconstructed by a user. Here, source colors 1330 may be selected manuallyby a user. Then, target colors 1325 may be manually mapped to the sourcecolors 1330 by the user. The manual mapping may be facilitated via thecomputer system.

In one aspect, the technology in the present disclosure may color agrayscale input target image by luminance-based probabilistic colortransfer. In another aspect, the system may automatically recolor acolor input target image by color-based probabilistic color transfer.This probabilistic color transfer may have two different versions fordifferent types of input target images, as shown in FIG. 7. Here, thecolor model of CIELαβ is utilized as an example. The color model may beother color models whose luminance and color component are independent.

The description of the process will be set forth below according to thefollowing outlines: (1) color-based probabilistic color transfer, and(2) luminance-based probabilistic color transfer.

The color-based probabilistic transfer may be based on the percentagevector matrix (P) calculated by the probability matrix generation. Tosolve the illumination distribution change of traditional color transfermethods, the color-based probabilistic color transfer may be applied inthe α and β plane of target image.

R j = ∑ 1 ≤ i ≤ K ∼  P i  ⊗ ~  ( σ ~ S j  ( i ) σ ~ T j i  ⊗ ~  (T j  ⊖ ~  μ ~ T j i )  ⊕ ~  μ ~ S j ) ( 14 )

Where, P_(i) is the i^(th) row in P. T_(j), S_(j) and R_(j) are thetarget, source and recolored image with j=α, β. {tilde over (μ)} and{tilde over (σ)} are the PLIP mean value and PLIP standard deviation ofeach region. For the L plane of the recolored image, it is just the sameas the one of the target image as shown in Eqn. (15).

R _(L) =T _(L)  (15)

For a grayscale input target image, the standard deviation informationof color components for each segment may not be important. Hence, thecolor-based probabilistic color transfer equation may be modified to bea luminance-based probabilistic color. As shown in Eqn. (16), theluminance-based probabilistic color may be processed using thepercentage vector matrix generated by image segmentation and the colormapping function (

(i)).

R j = { ∑ 1 ≤ i ≤ K ∼  P i  ⊗ ~  μ ~ S j , j = α , β T L , J = L ( 16)

Where, P_(i) is the i^(th) row in P. S_(j) and R_(j) are the source andrecolored image with j=α, β. For the L plane of the colored image, it isjust the same as the one of the target image as shown in Eqn. (16).

Referring to FIG. 7, a sub-method of FIG. 1 corresponding to performinga probabilistic color transfer is shown. Process block 1350 may befurther described. At process block 1355, the computer system maydetermine if the target image is a grayscale image. If the target imageis grayscale, a grayscale-based probabilistic color transfer may beimplemented at process block 1360. If the target image is not grayscale,a color-based probabilistic color transfer may be implemented at processblock 1365.

As discussed above, an inverse color model transformation may be appliedto transfer the output image into any color model, such as RGB, CIELαβand other color models. Referring to FIG. 8, a sub-method of FIG. 1corresponding to performing an inverse color model transformation isshown. Process block 1400 may be further described. At process block1405, the computer system may determine if the target image is agrayscale image. If the target image is grayscale, the computer systemmay transfer the output image to a default color model at process block1410. If the target image is not grayscale, the computer system maytransfer the output image to an original color model. The original colormodel may be associated with the input target image.

Several example applications are provided herein. The applications arenot intended to limit the present disclosure to any particular system ormethod.

FIG. 9 shows one non-limiting HIC-based color transfer system and methodaccording to the present disclosure. Using the system and method 1450according to the present disclosure, a source image may be utilized toextract the dominant source colors' information by using a color-basedimage segmentation method. This dominant source colors' information maybe further applied in the color-based probabilistic color transfer torecolor the input target image to generate recolored images.

Specifically, at process block 1455, a color source image is input.Similarly, at process block 1465, a color target image is input. Atprocess block 1460, color-based source image segmentation may occur. Atprocess block 1470, color-based target image segmentation may occur. Atprocess block 1475, a color map may be built. Then, at process block1480, color-based probabilistic color transfer may occur. The resultingrecolored image may be output at process block 1485.

FIG. 10 shows another non-limiting HIC-based color transfer system andmethod, using a source image, according to the present disclosure. Usingthe system and method 1500 according to the present disclosure, a sourceimage may be utilized to extract the dominant source colors' informationby using a color-based image segmentation method. This dominant sourcecolor information can be further applied in the luminance-basedprobabilistic color transfer to recolor the grayscale input target imageto generate colored images.

Specifically, at process block 1505, a color source image is input.Similarly, at process block 1515, a grayscale target image is input. Atprocess block 1510, color-based source image segmentation may occur. Atprocess block 1520, luminance-based target image segmentation may occur.At process block 1525, a color map may be built. Then, at process block1530, luminance-based probabilistic color transfer may occur. Theresulting recolored image may be output at process block 1535.

FIG. 11 shows another non-limiting HIC-based color transfer system forimage sequence generation. This probabilistic image sequence (PIS)system 1550 is an animation process, which may output an image sequenceshowing the gradually changing color. As an example, it may show thechange of animation of a tree image from winter, through spring, andfinally to autumn. This may be applied to many applications, includingcomputer animation and video editing. The first, middle (M^(th)) andlast image (N^(th)) in the output image sequence may have threedifferent ambiences from three different source images. The other imagesin the sequence will present the gradually changing color from oneambience to another ambience.

In this algorithm, after color-based image segmentation and colormapping,

,

,

and

,

,

are obtained from three different source images. To make the imagesequence have the effect of changing ambiences, the color variationcurve may be utilized to calculate the color information (μ_(S) _(j)^(i) and σ_(S) _(j) ^(i)) for probabilistic color transfer of each imagein the output image sequence. Here, the two linear color variation curveis shown, which is described in Eqns. (17)-(18).

μ ~ S j i = { μ ~ 2 2  ( i )  ⊖ ~  μ ~ 1 1  ( i ) M - 1  ( j  ⊖ ~ 1 )  ⊕ ~  μ ~ 1 1  ( i ) j = 1 , …  , M μ ~ 2 2  ( i )  ⊖ ~  μ~ 1 3  ( i ) M - N  ( j  ⊖ ~  M )  ⊕ ~  μ ~ 2 2  ( i ) j = M + 1, …  , N ( 17 ) σ ~ S j i = { σ ~ 2 2  ( i )  ⊖ ~  σ ~ 1 1  ( i )M - 1  ( j  ⊖ ~  1 )  ⊕ ~  σ ~ 1 1  ( i ) j = 1 , …  , M σ ~ 2 2 ( i )  ⊖ ~  σ ~ 1 3  ( i ) M - N  ( j  ⊖ ~  M )  ⊕ ~  σ ~ 2 2 ( i ) j = M + 1 , …  , N ( 18 )

After getting j^(th) color information, Eqns. (19)-(21) may be used toget the j^(th) image in the output image sequence after beingtransformed back to the RGB color plane.

$\begin{matrix}{{R_{L}(j)} = T_{L}} & (19) \\{{R_{\alpha}(j)} = {\sum\limits_{1 \leq i \leq K}^{\sim}{P_{i}\overset{\sim}{\otimes}\left( {{\frac{\sigma_{S^{j}}^{i}}{\sigma_{T_{\alpha}}^{i}}\overset{\sim}{\otimes}\left( {T_{\alpha}^{i}\overset{\sim}{\ominus}\mu_{T_{\alpha}}^{i}} \right)}\overset{\sim}{\oplus}\mu_{S^{j}}^{i}} \right)}}} & (20) \\{{R_{\beta}(j)} = {\sum\limits_{1 \leq i \leq K}^{\sim}{P_{i}\overset{\sim}{\otimes}\left( {{\frac{\sigma_{S^{j}}^{i}}{\sigma_{T_{\beta}}^{i}}\overset{\sim}{\otimes}\left( {T_{\beta}^{i}\overset{\sim}{\ominus}\mu_{T_{\beta}}^{i}} \right)}\overset{\sim}{\oplus}\mu_{S^{j}}^{i}} \right)}}} & (21)\end{matrix}$

FIG. 12 shows another non-limiting HIC-based color transfer system andmethod 1600 for fashion design. In the fashion industry, the designerneeds to pick the right color for their design. However, the colorpalette has only limited color types. Hence, the designers often onlyconsider the color in their minds or find a similar color in otherimages. Accordingly, they need a system and method to show their designin different colors which gradually change from some dominant colors.After generating a first design, a designer may input the image into acomputer with a camera as the target image. Then the system described inthe present disclosure can generate a sequence of images showing thecolor's gradual changes. The designer can then pick the color he/shewants.

FIG. 13 shows another non-limiting HIC-based color transfer system andmethod 1650 for supporting systems for people with color blindness. Thepresent disclosure may support the transfer of the images wherein somecolors cannot be distinguished by people with color blindness. As anexample, a camera positioned within a pair of glasses may collect theimages, and transfer some specific colors which cannot be recognized bypeople with color blindness into some other colors that can be seen bythese people, and then show these images on the glasses screen. This mayhelp people with color blindness or difficulty seeing the contentsclearly. However, this supporting system is not limited to glasses.

FIG. 14 shows another non-limiting HIC-based color transfer system andmethod 1700 for printing systems for people with color blindness. Whenthe computer sends the input image to a printer, the printer can providedifferent working models according to different requirements of users.If the user has the problem of red or green color blindness, the printercan utilize this system to do the color transfer and print the pictureswhich can be seen by people with red or green blindness (using sourcecolor input 1, 1720). If the user has the problem of blue colorblindness, the printer can also use this system with different sourcecolor settings to display the images (using source color input 2, 1740).The option of this system is not limited to these two kinds ofsituations—it may include other specific color blindness problems, andtherefore use corresponding source color settings. This system may helpthe printer to print the images which can be easily seen by people withthese specific types of color blindness or deficiencies.

In some aspects, this system can be applied in a display device totransfer the images wherein some colors cannot be distinguished bypeople with color blindness or deficiency to images which can be seen bythese people.

FIG. 15 shows another non-limiting HIC-based color transfer system andmethod 1750 for image coloring without a source image. Instead of usinga source image (as previously described), the system 1750 may utilize adeep neural network to generate the source color information (e.g. meanvalue and standard variance value) based on a structure feature of eachsegment in the target image.

In the training phase, color training images may be processed withluminance-based image segmentation to extract the color feature (e.g.mean value and standard variance value) and structure feature of eachsegment. Using the structure feature of each segment as and input, andthe color feature as the output, the deep neural network may be trainedto minimize cost function.

$\begin{matrix}{\underset{DNN}{\arg \mspace{14mu} \min}{\sum\limits_{j = 1}^{k\; 1}{\sum\limits_{i = 1}^{k\; 2}{{{{DNN}\left( {F_{S}\left( I_{j}^{i} \right)} \right)} - {F_{C}\left( I_{j}^{i} \right)}}}^{2}}}} & (22)\end{matrix}$

Where, F_(S)(I_(j) ^(i)) represents the structure feature of j^(th)segment in i^(th) image of color training image database. DNN( )represents the deep neural network. F_(C)(I_(j) ^(i)) represents thecolor feature (mean value and standard variance value) of j^(th)segmentation in i^(th) image of color training image database.

In the test phase, the grayscale target image may be processed withluminance-based image segmentation. The structure feature of eachsegmentation may be extracted and fed into the deep neural network togenerate the corresponding color feature information. Using the colorfeature information as source color information, luminance-basedprobabilistic color transfer may be applied to recolor the target image.

FIG. 16 shows another non-limiting HIC-based color transfer system andmethod 1800 for partial image coloring. As opposed to color transferringthe whole image (as previously described), a partial color transfersystem may achieve recoloring/coloring of only a portion of images (e.g.selected objects) and leave other portions unchanged. The desiredrecolored/colored parts of the image may be selected manually or by anobject detection algorithm. By only processing selected objects with aprobabilistic color transfer algorithm, local partial color transfer maybe achieved. This is distinct compared to the previous examples, whereglobal color transfer may occur.

FIG. 17 shows a non-limiting HIC-based color transfer system and method1850 for use with video. For each frame image in the video, a targetimage may be acquired. Accordingly, the HIC-based color transfer systemmay process a recoloring/coloring task to achieve video recoloring andcoloring.

Still referring to FIG. 17, at process block 1852, a target video may beused as an input to the system 1850. Next, a user may input dominantsource colors (via any of the methods previously described) at processblock 1854. The dominant source color numbers may then be determined atprocess block 1856. Iteration may be used to process each frame in thevideo, with each frame being processed as an image.

The iteration may be initialized by setting variable i to equal 1 atprocess block 1858. Decision block 1860 may then determine if i is lessthan or equal to the number of total frames in the target video. If i isless than or equal to the number of total frame in the target video, theith frame may be taken as an image at process block 1862. Then, atprocess block 1864, the image may be transformed via a color model. Theimage may then be segmented at process block 1866. Once segmented, acolor mapping may be determined for the dominant source colors and thedominant target colors, according to process block 1868. A probabilisticcolor transfer in the ith frame may then occur at process block 1870.Next, the inverse color model transform may be applied to get acorresponding ith output frame, according to process block 1872. Oncethe ith frame has been processed, i may be incremented by 1 at processblock 1874.

Referring again to decision block 1860, if i exceeds the number offrames in the target video, then a video output may be generated atprocess block 1876.

FIG. 18 shows a non-limiting HIC-based color transfer system and method1900 for use with video. Specifically, system/method 1900 shows specificobject color transfer for use in video editing. The examplesystem/method 1900 may have a specific object sample and a video asinputs.

By using objection detection algorithms with structural information(including texture information) to find the specific object sample inthe input video, the specific objects in each frame of the input videomay be recolored/colored by a corresponding probabilistic color transferalgorithm. As one non-limiting example, the system 1900 may help thefilm editor to track and recolor/color some specific objects in thewhole system to explore different art effects.

FIG. 19 shows a non-limiting HIC-based color transfer system and method1925 that may be used by product designers. In the product design, thephysical samples and virtual samples in the computer-aided designsoftware typically have some difference in color appearance. To achievecolor design without the cost of producing physical samples for eachcolor option, this system may be applied to take an image of an actualphysical sample and recolor it to different color appearances. This mayenable designers to select their preferred real-world color, as thecoloring is done on the image of the physical sample.

FIG. 20 shows a non-limiting HIC-based color transfer system and method1950 that may be used to assist people with a learning disorder ordyslexia. People with a learning disorder/dyslexia typically havereading problems. Research shows that color can assist theseindividuals. The proposed system may utilize a text detection method tofind the locations of all text in an image. Then, a probabilistic colortransfer may be used to recolor or color the text in the image to be ina color which may be easier to read.

In one aspect, the present disclosure may be utilized for camouflage(e.g. the military). To make the clothes closer to the background toreduce the possibility of exposure, people can take a picture of theenvironment. Using the color information extracted from these backgroundpictures, the system in the present disclosure may be utilized torecolor the camouflage model. The present disclosure can be used, butnot limited to, for camouflage in clothes, car, and other militaryfacilities.

In one aspect, the present disclosure can be utilized to recolor hair inimages to let users decide the color they want for their hair.

In another aspect, the present disclosure may be used for Whole SlidingImaging. This present disclosure may do the color standardization inwhole slide imaging to deal with the problem of color inconsistency inhistology images, resulting from the differences in slide thickness,staining, scanning parameters, and illumination. Using the presentdisclosure, the problem of similar objects in histology images showingdifferent color properties may be overcome. The present disclosure isnot limited to whole sliding imaging—it may be utilized for other typesof medical images.

In another aspect, the present disclosure may be used for housedecoration designs. As an example, when people want to repaint theirwall, he/she can take an image of their house and recolor the wall withdifferent colors. This helps users to pick the best color of the paintfor their house. This is not limited to wall painting—it may also beapplied furniture to decide color.

In another aspect, the present disclosure may be used for databasestandardization. For different color images in the database, databasestandardization is a vital preprocess to standardize the colordistribution. As an example, histology databases have the problem ofcolor inconsistency, resulting from the differences in operator, slidethickness, staining, scanning parameters, and illumination. To get astandardized database, the system in this present disclosure can beapplied to recolor each image in the database with the same source colorsetting (manually from color palette or automatic extraction from astandard color image as the source image). This system can be applied,but not limited to, other color-related database standardization tasks.

In another aspect, the present disclosure may be used for biometrics. Inthe performance test of face recognition, using the original face is anormal way to do simulation tests. However, in real applications,people's faces will have some changes, resulting from make-up,luminance, lighting condition, and other reasons. Hence, the robustnessof face recognition should be considered in real application of facerecognition. In the simulation test, it is difficult to take photos ofpeople's faces with different conditions. Using the system in thispresent disclosure, one original face can be modified to generatevarious modified faces to do the tests. This system can be applied, butnot limited to, faces, fingerprints, iris recognition, and otherbiometrics.

In another aspect, the present disclosure may be used for security. Asone non-limiting example, images taken at an undisclosed outdoorlocation may be segmented according to the present disclosure, and thedominant target colors may be compared to a database of known landscapeimages corresponding to dominant source colors. Using the mappingmethods disclosed, the undisclosed outdoor location may be determined.

In another aspect, the present disclosure includes a method oftransferring a grayscale target image into a colored target image in anRGB color model, by copying intensity values of the grayscale targetimage three times, each time to be one component of the three R, G, Bcomponents.

The sensors described with reference to the systems and methodsdescribed herein can be of any suitable type, such as CCD imagingsensors, CMOS imaging sensors, or any analog or digital imaging sensor.The sensors may be responsive to electromagnetic radiation outside thevisible spectrum, such as thermal, gamma, multi-spectral and x-raysensors. The sensors, in combination with other components in theimaging system, may generate a file in any format, such as the raw data,GIF, JPEG, TIFF, PBM, PGM, PPM, EPSF, X11 bitmap, Utah Raster ToolkitRLE, PDS/VICAR, Sun Rasterfile, BMP, PCX, PNG, IRIS RGB, XPM, Targa,XWD, PostScript, and PM formats on workstations and terminals runningthe X11 Window System or any image file suitable for import into thedata processing system. Additionally, the system may be employed forgenerating video images, such as digital video images in the .AVI, .WMV,.MOV, .RAM and .MPG formats.

The systems and methods described herein may be implemented in an imageprocessor which may include microcontrollers and microprocessorsprogrammed to receive data from the image sensor pixels and convert thedata into an RGB value for display on a monitor. The image processorsmay be configured with hardware and software to perform one or more ofthe methods, and any combination of the one or more methods, describedherein. The image processor may include a central processing unit (CPU),a memory, and an interconnect bus. The CPU may include a singlemicroprocessor or a plurality of microprocessors for configuring theimage processor as a multi-processor system. The memory may include amain memory and a read-only memory. The image processor may also includeone or more mass storage devices, e.g., any of various disk drives, tapedrives, FLASH drives, etc. The main memory can comprise dynamic randomaccess memory (DRAM) and/or high-speed cache memory. In operation, themain memory stores at least portions of instructions and data forexecution by a CPU.

The systems and methods may include a mass storage system, such as oneor more magnetic disk or tape drives or optical disk drives, for storingdata and instructions for use by the image processor. At least onecomponent of the mass storage system, possibly in the form of a diskdrive or tape drive, stores the database used for processing the signalsmeasured from the image sensors. The mass storage system may also (oralternatively) include one or more drives for various portable media,such as a floppy disk, a compact disc read-only memory (CD-ROM), DVD, oran integrated circuit non-volatile memory adapter (i.e. PC-MCIA adapter)to input and output data and code to and from the image processor.

The image processor may also include one or more input/output interfacesfor data communications. The data interface may be a modem, a networkcard, serial port, bus adapter, or any other suitable datacommunications mechanism for communicating with one or more local orremote systems. The data interface may provide a relatively high-speedlink to a network, such as the Internet. The communication link to thenetwork may be, for example, optical, wired, or wireless (e.g., viasatellite or cellular network). Alternatively, the image processor mayinclude a mainframe or other type of host computer system capable ofcommunicating via the network.

The image processor may also include suitable input/output ports or usethe interconnect bus for interconnection with other components, a localdisplay, and keyboard or other local user interface for programmingand/or data retrieval purposes.

In certain embodiments, the image processor includes circuitry for ananalog-to-digital converter and/or a digital-to-analog converter. Insuch embodiments, the analog-to-digital converter circuitry convertsanalog signals received at the sensors to digital signals for furtherprocessing by the image processor.

Certain components of the image processor are those typically found inimaging systems used for portable use as well as fixed use. In certainembodiments, the image processor may be a general purpose computersystem, e.g., of the types used as servers, workstations, personalcomputers, network terminals, and the like. Certain aspects of thesystems and methods described herein may relate to the softwareelements, such as the executable code and database for the serverfunctions of the image processor.

Generally, the methods and techniques described herein may be executedon a conventional data processing platform such as an IBM PC-compatiblecomputer running the Windows operating systems, a SUN workstationrunning a UNIX operating system or another equivalent personal computeror workstation. Alternatively, the data processing system may comprise adedicated processing system that includes an embedded programmable dataprocessing unit.

Certain embodiments of the systems and processes described herein mayalso be realized as software component operating on a conventional dataprocessing system such as a UNIX workstation. In such embodiments, theprocesses may be implemented as a computer program written in any ofseveral languages well-known to those of ordinary skill in the art, suchas (but not limited to) C, C++, FORTRAN, or Java. The processes may alsobe executed on commonly available clusters of processors, such asWestern Scientific Linux clusters, which may allow parallel execution ofall or some of the steps in the process.

Certain embodiments of the methods described herein may be performed ineither hardware, software, or any combination thereof, as those termsare currently known in the art. In particular, these methods may becarried out by software, firmware, or microcode operating on a computeror computers of any type, including pre-existing or already-installedimage processing facilities capable of supporting any or all of theprocessor's functions. Additionally, software embodying these methodsmay comprise computer instructions in any form (e.g., source code,object code, interpreted code, etc.) stored in any computer-readablemedium (e.g., ROM, RAM, magnetic media, punched tape or card, compactdisc (CD) in any form, DVD, etc.). Furthermore, such software may alsobe in the form of a computer data signal embodied in a carrier wave,such as that found within the well-known Web pages transferred amongdevices connected to the Internet. Accordingly, these methods andsystems are not limited to any particular platform, unless specificallystated otherwise in the present disclosure.

The systems described herein may include additional electronic,electrical and optical hardware and software elements for capturingimages without departing from the scope of the systems and methodsdescribed herein. For example, the system may include single-shotsystems, which in turn, may include one or more color filters coupledwith the imaging sensors (e.g., CCD or CMOS). In another embodiment, thesystem includes multi-shot systems in which the sensor may be exposed tolight from a scene in a sequence of three or more openings of the lensaperture. In such embodiments, one or more imaging sensors may becombined with one or more filters passed in front of the sensor insequence to obtain the additive color information. In other embodiments,the systems described herein may be combined with computer systems foroperating the lenses and/or sensors and processing captured images.

In one aspect, the present disclosure includes a method of transferringcolor to recolor a target image, the method including: a) receiving thetarget image, b) determining dominant source colors, c) transforming thetarget image into a color model including a target luminance componentand a target color information component, d) segmenting the target imageinto a plurality of target segments based on the target colorinformation component or the target luminance component, e) extractingdominant target colors from the target image by extracting informationfor at least one of the dominant target colors from each target segmentof the plurality of target segments, f) generating a color mappingrelationship between the dominant target colors and the dominant sourcecolors, and g) creating a recolored target image using the color mappingrelationship.

The method may further include receiving user input to alter thecreating of step g). The method may further include a source color inputalgorithm to generate the dominant source colors. The method may includewherein the dominant source colors are selected from a color palette.Additionally, the method may include extracting the dominant sourcecolors from the source image, wherein the source image is selected by auser. The method may further include wherein the target image comprisesa grayscale image. Additionally, the method may include wherein thecolor model comprises CIELαβ, YCbCr, a color model wherein the targetcolor information component and the target luminance component areindependent, or a combination thereof. The method may include whereinthe segmenting of step d) includes using an image segmentation algorithmto segment the target image, and calculating a possibility matrix foreach pixel belonging to each segment. The method may include wherein thesegmenting of step d) is based on the luminance component when thetarget image is a grayscale image. The method may include wherein thesegmenting of step d) is based on the target color information componentwhen the target image is a color image. The method may includecalculating the possibility matrix based on a distribution model,Gaussian distribution model, Binomial distribution model, Exponentialdistribution model, or Poisson distribution model. The method mayinclude calculating the possibility matrix based on a Beta distributionmodel, a Logit-normal distribution model, a Dirac delta function, or atruncated normal distribution model. The method may further includewherein the color mapping is manually input. The method may includewherein the creating of step g) comprises applying a grayscale-basedprobabilistic color transfer algorithm. The method may further includewherein the creating of step g) comprises applying a color-basedprobabilistic color transfer algorithm. The method may further includewherein all the dominant target colors in the target image aretransferred into more than one dominant source color. The method mayinclude wherein the determining dominant source colors of step b)includes: i) receiving a source image, ii) transforming the source imageinto a color model including a source luminance component and a sourcecolor information component, iii) segmenting the source image into aplurality of source segments based on the source color informationcomponent, and iv) extracting dominant source colors from the sourceimage by extracting information for at least one of the dominant sourcecolors from each source segment of the plurality of source segments.

In another aspect, the present disclosure includes a method for coloringan input grayscale image into an output color image. The methodincludes: a) selecting dominant source colors from a color palette orone color from a source image, b) applying a color model transformationto transform a target image in an original color model into a colormodel wherein a target luminance component and a target colorinformation component are independent, c) dividing the target image intoa plurality of target regions according to the target luminancecomponent, d) generating a color mapping relationship between at leastone dominant target color from each of the plurality of target regionsand a dominant source color, e) transferring dominant source colorinformation into a target image, and f) applying an inverse color modelalgorithm to transfer the color model to a selected color model.

The method may further include receiving user input to alter thetransferring of step f). The method may further include wherein thesource image is selected by a user. The method may further includewherein the color model comprises CIELαβ, YCbCr, a color model whereinthe target color information component and the target luminancecomponent are independent, or a combination thereof. The method mayfurther include wherein the color mapping is manually input. The methodmay further include wherein the transferring of step e) comprisesapplying a grayscale-based probabilistic color transfer algorithm.

In another aspect, the present disclosure includes a method forrecoloring an input image into an output image with another colorappearance, the method comprising: a) selecting dominant source colors,b) applying a color model transformation algorithm to transform a targetimage in an original color model into a color model wherein a targetluminance component and a target color information component areindependent, c) dividing the target image into a plurality of targetsegments according to the target color information component, d)extracting dominant target colors from the target image by extractinginformation for at least one of the dominant target colors from eachtarget segment of the plurality of target segments, e) generating acolor mapping relationship between the dominant target colors and thedominant source colors, f) transferring source color information into atarget image based on information generated from a source color inputalgorithm, and g) applying an inverse color model algorithm to transferthe color model to a selected color model.

The method may further include receiving user input to alter thetransferring of step f). The method may further include a source colorinput algorithm to generate the dominant source colors. The method mayfurther include wherein the dominant source colors are selected from acolor palette. The method may further include extracting the dominantsource colors from a source image, wherein the source image is selectedby a user. The method may further include wherein the color modelcomprises CIELαβ, YCbCr, a color model wherein the target colorinformation component and the target luminance component areindependent, or a combination thereof. The method may further includewherein the dividing of step c) includes using an image segmentationalgorithm to segment a source image, and calculating a possibilitymatrix for each pixel belonging to each segment. The method may furtherinclude further comprising calculating the possibility matrix based on adistribution model, Gaussian distribution model, Binomial distributionmodel, Exponential distribution model, or Poisson distribution model.The method may further include calculating the possibility matrix basedon a Beta distribution model, a Logit-normal distribution model, a Diracdelta function, or a truncated normal distribution model. The method mayfurther include wherein the color mapping is manually input. The methodmay further include wherein the transferring of step f) comprisesapplying a color-based probabilistic color transfer algorithm.

In another aspect of the present disclosure, a method for imagesegmentation by data grouping is included, the method comprising: a)receiving an original image, b) setting a number of segment groupsmanually or automatically via a computer algorithm, c) applying a colormodel transformation algorithm to transform the original image in anoriginal color model into a color model wherein a target luminancecomponent and a target color information component are independent, d)including the target color information component as a feature for eachpixel in the original image, e) grouping the pixels via a LogarithmicGMM method, using each target color information component.

The method may further include wherein the color model comprises CIELαβ,YCbCr, or other color models wherein the target color informationcomponent and the target luminance component are independent.

In another aspect, the present disclosure includes a method for imagesegmentation by data grouping, the method comprising: a) receiving anoriginal image, b) setting a number of segment groups manually orautomatically via a computer algorithm, c) applying a color modeltransformation algorithm to transform the original image in an originalcolor model into a color model wherein a target luminance component anda target color information component are independent, d) including thetarget color information component as a feature for each pixel in theoriginal image, and e) grouping the pixels via a Logarithmic K-meansmethod, using each target color information component.

The method may further include wherein the color model comprises CIELαβ,YCbCr, or other color models wherein the target color informationcomponent and the target luminance component are independent.

In another aspect, the present disclosure includes a method forgenerating an image sequence showing a gradual changing from a firstcolor appearance to a second color appearance, the method comprising: a)determining at least two sets of dominant source colors, b) applying acolor model transformation algorithm to transform a target image in anfirst color model into a color model wherein a target luminancecomponent and a target color information component are independent, c)segmenting the target image into a plurality of target segmentsaccording to the target color information component or the targetluminance component, d) extracting dominant target colors from thetarget image by extracting information for at least one of the dominanttarget colors from each target segment of the plurality of targetsegments, e) generating a color mapping relationship between thedominant target colors and the at least two sets of dominant sourcecolors, f) calculating color information for probabilistic colortransfer via a color variation curve, g) transferring the colorinformation into a target image by using the color information generatedfrom the color variation curve, and h) applying an inverse color modelalgorithm to transfer the first color model to a selected second colormodel.

The method may further include wherein the image sequence corresponds toa video animation.

In another aspect, the present disclosure includes a support system forcolor-impaired users, the system comprising: a pair of glassesconfigured to be worn by a color-impaired user, at least one cameraaffixed to the pair of glasses, and a processor in communication withthe at least one camera, the processor configured to: capture at leastone image via the at least one camera, determine dominant source colors,segment a target image into a plurality of target segments based on atarget color information component, extract dominant target colors fromthe target image by extracting information for at least one of thedominant target colors from each target segment of the plurality oftarget segments, generate a color mapping relationship between thedominant target colors and the dominant source colors, transfer colorinformation into the target image, generate images for thecolor-impaired user, and display the generated images on at least onelens of the pair of glasses.

In another aspect, the present disclosure includes a method for testingthe performance of biometrics recognition technology, the methodcomprising: a) receiving a biometrics image, b) determining dominantsource colors, c) segmenting the biometrics image into a plurality ofbiometric segments based on a biometrics color information component, d)extracting dominant target colors from the biometric image by extractinginformation for at least one of the dominant target colors from eachbiometric segment of the plurality of biometric segments, e) generatinga color mapping relationship between the dominant target colors and thedominant source colors, f) transferring color information into thebiometrics image, g) extracting at least one biometrics feature from thebiometrics image of step f), h) comparing the at least one biometricsfeature with a reference data set, and i) generating a test result.

The method may further include wherein the biometrics image is a singleface image. The method may further include wherein the at least onebiometrics feature corresponds to at least one of a face, a fingerprint,and an iris.

In another aspect, the present disclosure includes a method for coloringan input grayscale image into an output colorful image, the methodcomprising: a) applying a color model transformation algorithm totransform a target image in an original color model into a color modelwherein a target luminance component and a target color informationcomponent are independent, b) segmenting a target image into a pluralityof target segments based on the target luminance component, c)extracting structure features from the target image by extractinginformation for at least one of the structure features from each targetsegment of the plurality of target segments, d) generating a sourcecolor for each target segment based on each structure feature, via amachine learning model, e) transferring the dominant source colors intothe target image via a copy process, and f) applying an inverse colormodel algorithm to transfer the original color model to a selectedsecond color model.

The method may further include wherein the machine learning model is oneof a deep neural network or a neural network.

In another aspect, the present disclosure includes a method for partialcolor transfer, the method comprising: a) selecting an object to becolor transferred, the object included in a target image, b) determiningdominant source colors, c) transforming the target image from anoriginal color model into a color model including a target luminancecomponent and a target color information component, d) segmenting theobject into a plurality of object segments based on the target colorinformation component or the target luminance component, e) extractingdominant target colors from the object by extracting information for atleast one of the dominant target colors from each object segment of theplurality of object segments, f) generating a color mapping relationshipbetween the dominant target colors and the dominant source colors, g)transferring color information into the object, and h) applying aninverse color model algorithm to transfer the original color model to aselected color model.

The method may further include wherein the target image is in grayscale,and step g) includes a luminance-based probabilistic color transfer. Themethod may further include wherein the object to color transfer isselected manually. The method may further include wherein the object tocolor transfer is automatically detected via an object detectionalgorithm.

In another aspect, the present disclosure includes a method for partialcolor transfer in a video, the method comprising: a) inputting at leastone object to be color transferred, the at least one object included ina video, b) detecting the at least one object in each frame image of thevideo, c) determining dominant source colors, d) transforming each frameimage from an original color model into a color model including a frameluminance component and a frame color information component, e)segmenting the at least one object into a plurality of object segmentsbased on the frame color information component or the frame luminancecomponent, f) extracting dominant target colors from the at least oneobject by extracting information for at least one of the dominant targetcolors from each object segment of the plurality of object segments, g)generating a color mapping relationship between the dominant targetcolors and the dominant source colors, h) transferring color informationinto the at least one object in each frame image, and i) applying aninverse color model algorithm to transfer the original color model to aselected color model.

The method may further include wherein the video is grayscale and stepe) is based on the luminance component. The method may further includewherein the target image is a picture of a physical product sample.

In another aspect, the present disclosure includes a method ofrecoloring text for people with a learning disability, the methodcomprising: a) detecting and extracting text from a target image, b)transforming the target image from an original color model into a colormodel including a target luminance component and a target colorinformation component, c) determining dominant source colors, d)transferring color information into the text via probabilistic colortransfer, and e) applying an inverse color model algorithm to transferthe original color model to a selected color model.

The method may further include wherein the dominant source colors areselected from a color palette. The method may further include whereinthe dominant source colors are determined from a source image. Themethod may further include wherein step d) includes transferring viaprobabilistic color transfer. The method may further include wherein theselected color model is selected by a user. The method may furtherinclude wherein the text of step a) is in grayscale.

In another aspect, the present disclosure includes a non-transitorycomputer-readable medium having stored thereon instructions that, whenexecuted by a processor, cause the processor to execute the method ofany of the previous claims.

The foregoing merely illustrates the principles of the disclosure.Various modifications and alterations to the described aspects will beapparent to those skilled in the art in view of the teachings herein. Itwill thus be appreciated that those skilled in the art will be able todevise numerous systems, arrangements and methods which, although notexplicitly shown or described herein, embody the principles of thedisclosure and are thus within the spirit and scope of the presentdisclosure. Further, the exemplary aspects described herein can operatetogether with one another and interchangeably therewith. In addition, tothe extent that the prior art knowledge has not been explicitlyincorporated by reference herein above, it is explicitly beingincorporated herein in its entirety. All publications referenced hereinabove are incorporated herein by reference in their entireties.

We claim:
 1. A method of transferring color to recolor a target image,the method comprising: a) receiving the target image; b) determiningdominant source colors; c) transforming the target image into a colormodel including a target luminance component and a target colorinformation component; d) segmenting the target image into a pluralityof target segments based on the target color information component orthe target luminance component; e) extracting dominant target colorsfrom the target image by extracting information for at least one of thedominant target colors from each target segment of the plurality oftarget segments; f) generating a color mapping relationship between thedominant target colors and the dominant source colors; and g) creating arecolored target image using the color mapping relationship.
 2. Themethod of claim 1, further comprising receiving user input to alter thecreating of step g).
 3. The method of any of the previous claims,further comprising a source color input algorithm to generate thedominant source colors.
 4. The method of any of claims 1 and 2, whereinthe dominant source colors are selected from a color palette.
 5. Themethod of any of claims 1 and 2, further comprising extracting thedominant source colors from a source image, wherein the source image isselected by a user.
 6. The method of any of the previous claims, whereinthe target image comprises a grayscale image.
 7. The method of any ofthe previous claims, wherein the color model comprises CIELαβ, YCbCr, acolor model wherein the target color information component and thetarget luminance component are independent, or a combination thereof. 8.The method of any of the previous claims, wherein the segmenting of stepd) includes using an image segmentation algorithm to segment the targetimage, and calculating a possibility matrix for each pixel belonging toeach segment.
 9. The method of any of the previous claims, wherein thesegmenting of step d) is based on the luminance component when thetarget image is a grayscale image.
 10. The method of any of claims 1 to8, wherein the segmenting of step d) is based on the target colorinformation component when the target image is a color image.
 11. Themethod of claim 8, further comprising calculating the possibility matrixbased on a distribution model, Gaussian distribution model, Binomialdistribution model, Exponential distribution model, or Poissondistribution model.
 12. The method of claim 8, further comprisingcalculating the possibility matrix based on a Beta distribution model, aLogit-normal distribution model, a Dirac delta function, or a truncatednormal distribution model.
 13. The method of any of the previous claims,wherein the color mapping is manually input.
 14. The method of any ofthe previous claims, wherein the creating of step g) comprises applyinga grayscale-based probabilistic color transfer algorithm.
 15. The methodof any of claims 1 to 13, wherein the creating of step g) comprisesapplying a color-based probabilistic color transfer algorithm.
 16. Themethod of claim 15, wherein all the dominant target colors in the targetimage are transferred into more than one dominant source color.
 17. Themethod of claim 1, wherein the determining dominant source colors ofstep b) includes: i) receiving a source image; ii) transforming thesource image into a color model including a source luminance componentand a source color information component; iii) segmenting the sourceimage into a plurality of source segments based on the source colorinformation component; and iv) extracting dominant source colors fromthe source image by extracting information for at least one of thedominant source colors from each source segment of the plurality ofsource segments.
 18. A method for coloring an input grayscale image intoan output color image, the method comprising: a) selecting dominantsource colors from a color palette or one color from a source image; b)applying a color model transformation to transform a target image in anoriginal color model into a color model wherein a target luminancecomponent and a target color information component are independent; c)dividing the target image into a plurality of target regions accordingto the target luminance component; d) generating a color mappingrelationship between at least one dominant target color from each of theplurality of target regions and a dominant source color; e) transferringdominant source color information into a target image; and f) applyingan inverse color model algorithm to transfer the color model to aselected color model.
 19. The method of claim 18, further comprisingreceiving user input to alter the transferring of step f).
 20. Themethod of any of claims 18 to 19, wherein the source image is selectedby a user.
 21. The method of any of claims 18 to 20, wherein the colormodel comprises CIELαβ, YCbCr, a color model wherein the target colorinformation component and the target luminance component areindependent, or a combination thereof.
 22. The method of any of claims18 to 21, wherein the color mapping is manually input.
 23. The method ofany of claims 18 to 22, wherein the transferring of step e) comprisesapplying a grayscale-based probabilistic color transfer algorithm.
 24. Amethod for recoloring an input image into an output image with anothercolor appearance, the method comprising: a) selecting dominant sourcecolors; b) applying a color model transformation algorithm to transforma target image in an original color model into a color model wherein atarget luminance component and a target color information component areindependent; c) dividing the target image into a plurality of targetsegments according to the target color information component; d)extracting dominant target colors from the target image by extractinginformation for at least one of the dominant target colors from eachtarget segment of the plurality of target segments; e) generating acolor mapping relationship between the dominant target colors and thedominant source colors; f) transferring source color information into atarget image based on information generated from a source color inputalgorithm; and g) applying an inverse color model algorithm to transferthe color model to a selected color model.
 25. The method of claim 24,further comprising receiving user input to alter the transferring ofstep f).
 26. The method of any of claims 24 to 25, further comprising asource color input algorithm to generate the dominant source colors. 27.The method of any of claims 24 to 25, wherein the dominant source colorsare selected from a color palette.
 28. The method of any of claims 24 to25, further comprising extracting the dominant source colors from asource image, wherein the source image is selected by a user.
 29. Themethod of any of claims 24 to 28, wherein the color model comprisesCIELαβ, YCbCr, a color model wherein the target color informationcomponent and the target luminance component are independent, or acombination thereof.
 30. The method of any of claims 24 to 29, whereinthe dividing of step c) includes using an image segmentation algorithmto segment a source image, and calculating a possibility matrix for eachpixel belonging to each segment.
 31. The method of claim 30, furthercomprising calculating the possibility matrix based on a distributionmodel, Gaussian distribution model, Binomial distribution model,Exponential distribution model, or Poisson distribution model.
 32. Themethod of claim 30, further comprising calculating the possibilitymatrix based on a Beta distribution model, a Logit-normal distributionmodel, a Dirac delta function, or a truncated normal distribution model.33. The method of any of claims 24 to 32, wherein the color mapping ismanually input.
 34. The method of any of claims 1 to 33, wherein thetransferring of step f) comprises applying a color-based probabilisticcolor transfer algorithm.
 35. A method for image segmentation by datagrouping, the method comprising: a) receiving an original image; b)setting a number of segment groups manually or automatically via acomputer algorithm; c) applying a color model transformation algorithmto transform the original image in an original color model into a colormodel wherein a target luminance component and a target colorinformation component are independent; d) including the target colorinformation component as a feature for each pixel in the original image;and e) grouping the pixels via a Logarithmic GMM method, using eachtarget color information component.
 36. The method of claim 35, whereinthe color model comprises CIELαβ, YCbCr, or other color models whereinthe target color information component and the target luminancecomponent are independent.
 37. A method for image segmentation by datagrouping, the method comprising: a) receiving an original image; b)setting a number of segment groups manually or automatically via acomputer algorithm; c) applying a color model transformation algorithmto transform the original image in an original color model into a colormodel wherein a target luminance component and a target colorinformation component are independent; d) including the target colorinformation component as a feature for each pixel in the original image;and e) grouping the pixels via a Logarithmic K-means method, using eachtarget color information component.
 38. The method of claim 37, whereinthe color model comprises CIELαβ, YCbCr, or other color models whereinthe target color information component and the target luminancecomponent are independent.
 39. A method for generating an image sequenceshowing a gradual changing from a first color appearance to a secondcolor appearance, the method comprising: a) determining at least twosets of dominant source colors; b) applying a color model transformationalgorithm to transform a target image in an first color model into acolor model wherein a target luminance component and a target colorinformation component are independent; c) segmenting the target imageinto a plurality of target segments according to the target colorinformation component or the target luminance component; d) extractingdominant target colors from the target image by extracting informationfor at least one of the dominant target colors from each target segmentof the plurality of target segments; e) generating a color mappingrelationship between the dominant target colors and the at least twosets of dominant source colors; f) calculating color information forprobabilistic color transfer via a color variation curve with these atleast two sets of dominant source color; g) transferring the colorinformation into a target image by using the color information generatedfrom the color variation curve; and h) applying an inverse color modelalgorithm to transfer the first color model to a selected second colormodel.
 40. The method of claim 39, wherein the image sequencecorresponds to a video animation.
 41. A support system forcolor-impaired users, the system comprising: a pair of glassesconfigured to be worn by a color-impaired user; at least one cameraaffixed to the pair of glasses; a processor in communication with the atleast one camera, the processor configured to: capture at least oneimage via the at least one camera; determine dominant source colors;segment a target image into a plurality of target segments based on atarget color information component; extract dominant target colors fromthe target image by extracting information for at least one of thedominant target colors from each target segment of the plurality oftarget segments; generate a color mapping relationship between thedominant target colors and the dominant source colors; transfer colorinformation into the target image; generate images for thecolor-impaired user; and display the generated images on at least onelens of the pair of glasses.
 42. A method for testing the performance ofbiometrics recognition technology, the method comprising: a) receiving abiometrics image; b) determining dominant source colors; c) segmentingthe biometrics image into a plurality of biometric segments based on abiometrics color information component; d) extracting dominant targetcolors from the biometric image by extracting information for at leastone of the dominant target colors from each biometric segment of theplurality of biometric segments; e) generating a color mappingrelationship between the dominant target colors and the dominant sourcecolors; f) transferring color information into the biometrics image; g)extracting at least one biometrics feature from the biometrics image ofstep f); h) comparing the at least one biometrics feature with areference data set; and i) generating a test result.
 43. The method ofclaim 42, wherein the biometrics image is a single face image.
 44. Themethod of claim 42, wherein the at least one biometrics featurecorresponds to at least one of a face, a fingerprint, and an iris.
 45. Amethod for coloring an input grayscale image into an output colorfulimage, the method comprising: a) applying a color model transformationalgorithm to transform a target image in an original color model into acolor model wherein a target luminance component and a target colorinformation component are independent; b) segmenting a target image intoa plurality of target segments based on the target luminance component;c) extracting structure features from the target image by extractinginformation for at least one of the structure features from each targetsegment of the plurality of target segments; d) generating a sourcecolor for each target segment based on each structure feature, via amachine learning model; e) transferring the dominant source colors intothe target image via a copy process; and f) applying an inverse colormodel algorithm to transfer the original color model to a selectedsecond color model.
 46. The method of claim 45, wherein the machinelearning model is one of a deep neural network or a neural network. 47.A method for partial color transfer, the method comprising: a) selectingan object to be color transferred, the object included in a targetimage; b) determining dominant source colors; c) transforming the targetimage from an original color model into a color model including a targetluminance component and a target color information component; d)segmenting the object into a plurality of object segments based on thetarget color information component or the target luminance component; e)extracting dominant target colors from the object by extractinginformation for at least one of the dominant target colors from eachobject segment of the plurality of object segments; f) generating acolor mapping relationship between the dominant target colors and thedominant source colors; g) transferring color information into theobject; and h) applying an inverse color model algorithm to transfer theoriginal color model to a selected color model.
 48. The method of claim47, wherein the target image is in grayscale, and step g) includes aluminance-based probabilistic color transfer.
 49. The method of any ofclaims 47 to 48, wherein the object to color transfer is selectedmanually.
 50. The method of any of claims 47 to 48, wherein the objectto color transfer is automatically detected via an object detectionalgorithm.
 51. A method for partial color transfer in a video, themethod comprising: a) inputting at least one object to be colortransferred, the at least one object included in a video; b) detectingthe at least one object in each frame image of the video; c) determiningdominant source colors; d) transforming each frame image from anoriginal color model into a color model including a frame luminancecomponent and a frame color information component; e) segmenting the atleast one object into a plurality of object segments based on the framecolor information component or the frame luminance component; f)extracting dominant target colors from the at least one object byextracting information for at least one of the dominant target colorsfrom each object segment of the plurality of object segments; g)generating a color mapping relationship between the dominant targetcolors and the dominant source colors; h) transferring color informationinto the at least one object in each frame image; and i) applying aninverse color model algorithm to transfer the original color model to aselected color model.
 52. The method of claim 51, wherein the video isgrayscale and step e) is based on the luminance component.
 53. Themethod of claim 1 or 39, wherein the target image is a picture of aphysical product sample.
 54. A method of recoloring text for people witha learning disability, the method comprising: a) detecting andextracting text from a target image; b) transforming the target imagefrom an original color model into a color model including a targetluminance component and a target color information component; c)determining dominant source colors; d) transferring color informationinto the text via probabilistic color transfer; and e) applying aninverse color model algorithm to transfer the original color model to aselected color model.
 55. The method of claim 54, wherein the dominantsource colors are selected from a color palette.
 56. The method of claim54, wherein the dominant source colors are determined from a sourceimage.
 57. The method of any of claims 54 to 56, wherein step d)includes transferring via probabilistic color transfer.
 58. The methodof any of claims 54 to 57, wherein the selected color model is selectedby a user.
 59. The method of any of claims 54 to 58, wherein the text ofstep a) is in grayscale.
 60. A non-transitory computer-readable mediumhaving stored thereon instructions that, when executed by a processor,cause the processor to execute the method of any of the previous claims.